Differencing in a data replication appliance

ABSTRACT

Methods and apparatus are provided for copying data from a primary storage facility to a secondary storage facility which reduce the workload on the storage controller in the primary facility and minimize bandwidth usage. The primary storage facility includes a primary data replication appliance which transfers data to a secondary replication appliance. Updated data from a host is both stored through a storage controller in the primary facility and also received by the primary replication appliance. Logic in the primary replication appliance determines whether the immediately previous version of the data is in a buffer from a previous storage operation. If so, the current (updated) version of the data is compared with the previous version and the difference, such as calculated through a bit-wise exclusive-OR operation, is transferred to the secondary replication appliance. The process is reversed in the secondary replication appliance and the recreated updated version of the data stored in the secondary facility.

TECHNICAL FIELD

The present invention relates generally to transferring data over a datalink and, in particular, to applying a differencing algorithm to reducedata link bandwidth usage during data transfers.

BACKGROUND ART

Remote copying of data is an integral part of disaster recovery forprotecting critical data from loss and providing continuous dataavailability. In a disaster recovery support system, data write updatesto a primary or central data store are reproduced at a secondary, remotesite. The remote site is typically located at a distance from theprimary data store if protection from natural disasters is a concern,but may be adjacent to the primary site if equipment failure is the mainconcern. In the event of a failure at the primary data store, the remotesite can take over all operations, including data write updates, withconfidence that no data has been lost. Later, after repair, the primarydata store can be restored to the condition of the remote site and canresume all operations, including data write operations.

During remote copying, typically same-sized blocks of data are sent fromthe primary data store to the remote data site. In this way, data writeupdates at the primary data store are reproduced at the remote site soas to permit reconstruction of the data, including reconstruction of theexact sequence of data write updates that took place at the primary datastore. This reproducibility can be especially important, for example, ina banking system or other transaction log system. Thus, data writeupdates at the primary data store are collected and are periodicallysent to the remote site in a remote copy operation.

The various types of remote copy can require enormous amounts ofbandwidth over the data lines between the primary data store and theremote site controller. For example, if a primary data store controllercan support 20,000 input/output (I/O) operations per second, and if 50%of these operations are write operations, then the controller can handle10,000 write operations per second. If each write update involves 4 Kbytes, then bandwidth of 40 MB per second is required between theprimary controller and the remote site controller. This is a significantamount of bandwidth to provide, given currently available pricing fordata lines. Even though asynchronous remote copy can speed up writeupdates, it does not decrease the amount of bandwidth required.

One proposed system which addresses the issue of bandwidth usage ispresented in U.S. Pat. No. 6,327,671 entitled DELTA COMPRESSEDASYNCHRONOUS REMOTE COPY and assigned to the assignee of the presentapplication (“the '671 patent”), which patent is incorporated herein byreference in its entirety. As illustrated in FIG. 1, the systemdisclosed therein provides a remote copy operation that copies datawrite updates from a primary data store to a secondary data store byidentifying which bytes have changed and sending only the changed bytesfrom the primary data store to the secondary site. A data operation suchas an exclusive-OR (XOR) logic operation can be used to identify thechanged bytes. Many data storage systems include XOR facilities as partof their normal configuration, including systems that implement thewell-known RAID-type data storage. The XOR operation is used in the '671patent on the write updated block of data to be copied. Data compressioncan then be used on the XOR data block to delete the unchanged bytes,and then only the changed bytes are sent to the remote site. Thisreduces the amount of data being sent between the primary data store andthe remote site, and reduces the bandwidth required between the sites.In this way, the remote copy system is said to provide remote copyingwithout requiring a great deal of expensive bandwidth.

SUMMARY OF THE INVENTION

The present invention provides methods and apparatus for copying datafrom a primary storage facility to a secondary storage facility whichreduce the workload on the storage controller in the primary facilityand minimize bandwidth usage. The primary storage facility includes aprimary data replication appliance which transfers data to a secondaryreplication appliance. Updated data from a host is both stored through astorage controller in the primary facility and also received by theprimary replication appliance. Logic in the primary replicationappliance determines whether the immediately previous version of thedata is in a buffer from a prior storage operation. If so, the current(updated) version of the data is compared with the previous version andthe difference, such as calculated through a bit-wise exclusive-ORoperation, is compressed and transferred to the secondary replicationappliance. The process is reversed in the secondary replicationappliance and the recreated updated version of the data stored in thesecondary facility.

While the primary appliance is performing these operations, the storagecontroller may be performing other tasks, including allowing access bythe host. Therefore, the operations of the present invention, beingperformed in the primary replication appliance, do not adversely affectthe workload of the storage controller or of the host. Additionally,because the difference between the previous version of the data and thecurrent version, representing only bits which have changed, is typicallyhighly compressible, when it is compressed and transmitted to thesecondary replication appliance, bandwidth usage is reduced.

Other features and advantages of the present invention should beapparent from the following description of the preferred embodiments,which illustrates, by way of example, the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a prior art remote copy system;

FIG. 2 is a block diagram of a data replication system of the presentinvention;

FIG. 3 is a flowchart of the operational steps performed by the primaryreplication appliance of the present invention; and

FIG. 4 is a flowchart of the operational steps performed by thesecondary replication appliance of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 2 is a block diagram of a data replication system 200 of thepresent invention. The system 200 includes a primary data facility 210and a secondary data facility 250. Both facilities 210 and 250 include astorage controller 212, 252 coupled to a data storage device or array214, 254. Both facilities 210, 250 of the present invention furtherinclude a replication appliance 220, 260, interconnected to each otherby way of appropriate interfaces 221, 261 directly or through a network,represented by the connection 202. The storage controller 212 and thereplication appliance 220 of the primary data facility 210 areoperatively coupled through respective interfaces 213, 223 to a hostdevice 204.

The primary and secondary replication appliances 210, 260 both include:a processor 222, 262; a memory 224, 264 for storage of softwareinstructions to be executed on the processor 222, 262; disk storage 226,266; and a first-in-first-out (FIFO) buffer 228, 268 for temporarystorage of data. The appliances 220, 260 also include disk storage 228,268. Logic 230, 270 for performing the operations of the presentinvention may reside in the memory 224, 264, in firmware or in aseparate processor.

Referring to the flowchart of FIG. 3, in operation the host 204 sendsdata to the primary storage controller 212 to be stored in the primarystorage device 214 (step 300). The primary replication appliance 220also receives the data (step 302), either simultaneously from the host204 or forwarded by the primary controller 210, and ultimately sends acopy of the data over the connection 202 (step 306) for duplicatestorage by the secondary replication appliance 260 (step 308). Duringthe process, the primary appliance 220 temporarily stores the data inthe buffer 226 (step 304). As other data is received from the host 204,it is also temporarily stored in the buffer 226. When an update topreviously stored data is transmitted by the host 204 (step 310), it isboth stored by the primary storage controller 212 (step 312) andreceived by the primary replication appliance 220 (step 314). Under thedirection of the logic 230, the primary appliance 220 determines whetherthe immediately previous version of the data being updated is still inthe buffer 226 (step 316). Depending on the size of the buffer 226 andthe amount of time that has passed since the previous version of thedata was received, the previous version of the data may not yet havebeen overwritten in the buffer 226 by other, more recent data. If theprevious version of the data is no longer present, the logic 230 directsthat the current version be transferred to the secondary replicationappliance 260 (step 318).

If the previous version of the data being updated is still present, thelogic 230 directs that it be compared with the current (updated) version(step 320). Because of the easy availability and reversibility of anexclusive-OR (XOR) function, preferably a bit-wise XOR operation isperformed on the two sets of data to calculate a “difference” (step322). Other mathematical transformations, or sets of complementarytransformations, may be substituted for the XOR operation. Anappropriate other set of two transformations includes a firsttransformation T1 which is performed on the current version of the dataand the immediately previous version of the data to generate a newversion D. A second, complementary, transformation T2 is subsequentlyapplied to the new version D and the immediately previous version togenerate (recreate) the current version. A further property of thesecond transformation T2 is that, when it is applied to the new versionD and the current version, the immediately previous version should begenerated (recreated). When the bit-wise XOR operation is used, bothtransformations T1 and T2 are the same. As used herein, the term“difference” will refer to the new version D which is the result of anysuch transformation T1.

Once generated or calculated, the difference is compressed (step 324) toreduce its size. Subsequently, the compressed difference is transferredto the secondary replication appliance 260 (step 328). While the primaryappliance 220 is performing these operations, the primary storagecontroller 212 may be performing other tasks, including allowing accessby the host 204. Therefore, the operations of the present invention,being performed in the replication appliance 220, do not adverselyaffect the workload of the primary storage controller 212 or of the host204. Additionally, because the typically highly compressible differencebetween the previous version of the data and the current version,representing only bits which have changed, is compressed and transmittedto the secondary replication appliance 260, bandwidth usage is reduced.

At the direction of the logic 230, the difference may be encrypted toimprove security before being transferred (step 326). If the previousversion of the data was not still in the buffer 226 and the unprocessedcurrent version is to be transferred to the secondary appliance 260, ittoo may be compressed, encrypted or both before being transferred.

Referring to the flowchart of FIG. 4, when the difference is received bythe secondary replication appliance 260 (step 400), it is decrypted(step 402) if necessary and decompressed (step 404). The secondaryappliance, under the direction of the logic 270, determines whether theprevious version of the data is still in the buffer 266 (step 406). Ifso, the logic directs that the difference and the previous version ofthe data be compared (step 408), again preferably using a bit-wise XORto reverse the previous process and recreate the current (updated)version of the data (step 410). The recreated current version of thedata is then stored, through the secondary storage controller 252, inthe secondary storage 254 (step 412).

If the previous version of the data is not still in the buffer 266 ofthe secondary replication appliance 260, it must be retrieved from thesecondary storage 254 (step 414) before it can be compared with thedifference (step 408) to recreate the current version (step 410) and bestored (step 412). If the secondary appliance employs aread_before_write function when storing data, the previous version ofthe data will first be read from the storage 254 and temporarily storedin a nonvolatile memory, which may include the buffer 266. Only afterthe previous version is hardened in this fashion will the recreatedcurrent version overwrite the previous version in the secondary storage254. Thus, if the overwrite fails, the previous version is stillavailable for recovery. Consequently, no extra step is required by thesecondary appliance 260 to obtain the previous version of the databefore recreating the current version.

In addition to the savings in bandwidth, the present inventionbeneficially permits the development and deployment of one logic todirect the process described herein, regardless of what type of storagecontroller the replication appliance is used with.

It is important to note that while the present invention has beendescribed in the context of a fully functioning system, those ofordinary skill in the art will appreciate that the processes of thepresent invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies regardless of the particular type ofsignal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media suchas a floppy disk, a hard disk drive, a RAM, and CD-ROMs andtransmission-type media such as digital and analog communication links.

The description of the present invention has been presented for purposesof illustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated. Moreover, although described above withrespect to methods and systems, the need in the art may also be met witha computer program product containing instructions for copying updateddata from a primary storage facility to a secondary storage facility ora method for deploying computing infrastructure comprising integratingcomputer readable code into a computing system for performing the same.

1. A method for copying updated data from a primary storage facility toa secondary storage facility, comprising: receiving a current version ofdata in a primary data replication appliance from a host substantiallysimultaneously with receipt of the current version of the data in aprimary storage controller; determining if an immediately previousversion of the data is stored in a first buffer in the primary datareplication appliance; if the immediately previous version of the datais stored in the first buffer: calculating a difference between thecurrent version of the data and the immediately previous version of thedata; and transferring the calculated difference from the primary datareplication appliance to a secondary data replication appliance; and ifthe immediately previous version of the data is not stored in the firstbuffer, transferring the current version of the data to the secondarydata replication appliance.
 2. The method of claim 1, furthercomprising: receiving the difference from the primary data replicationappliance at the secondary data replication appliance; determining if animmediately previous version of the data is stored in a second buffer inthe secondary data replication appliance; if the immediately previousversion of the data is stored in the second buffer, reading theimmediately previous version of the data from the second buffer; if theimmediately previous version of the data is not stored in the secondbuffer, reading the immediately previous version of the data fromstorage at the secondary data replication appliance; recreating thecurrent version of the data in the secondary data replication appliancefrom the immediately previous version of the data and the receiveddifference; and storing the current version of the data at a secondarystorage controller attached to the secondary data replication appliance.3. The method of claim 1, wherein calculating the difference between thecurrent version of the data and the immediately previous version of thedata comprises applying a bitwise exclusive-OR to the current versionand the immediately previous version.
 4. The method of claim 1, furthercomprising: receiving the calculated difference from the primary datareplication appliance at the secondary data replication appliance;reading the immediately previous version of the data from storage at thesecondary data replication appliance; storing the immediately previousversion of the data in a non-volatile memory in the secondary datareplication appliance; calculating the current version of the data fromthe immediately previous version of the data and the receiveddifference; and storing the current version of the data at a secondarystorage controller attached to the secondary data replication appliance.5. The method of claim 4, wherein storing the current version of thedata comprises overwriting the immediately previous version of the data.6. The method of claim 4, wherein: calculating the difference betweenthe current version of the data and the immediately previous version ofthe data comprises applying a bitwise exclusive-OR to the currentversion of the data and the immediately previous version of the data;and calculating the current version of the data from the immediatelyprevious version of the data and the received difference comprisesapplying a bitwise exclusive-OR to the difference and the immediatelyprevious version of the data.
 7. The method of claim 1, furthercomprising compressing the difference before transferring the differenceto the secondary data replication appliance.
 8. The method of claim 1,further comprising encrypting the difference before transferring thedifference to the secondary data replication appliance.
 9. A primarydata replication appliance, comprising: a first interface for receivinga current version of the data from a host device, the current version ofthe data being received substantially simultaneously in a primarystorage controller; a first buffer for temporary storage of the currentversion of the data; means for determining if an immediately previousversion of the current version of the data is stored in the firstbuffer; logic for calculating a difference between the current versionof the data and the immediately previous version of the data; and asecond interface for transferring the calculated difference to asecondary data replication appliance.
 10. The appliance of claim 9,wherein the calculating logic comprises logic for performing a bitwiseexclusive-OR on the current version of the data and the immediatelyprevious version of the data.
 11. The appliance of claim 9, furthercomprising means for compressing the difference before transferring thecalculated difference to the secondary data replication appliance. 12.The appliance of claim 9, further comprising means for encrypting thedifference before transferring the calculated difference to thesecondary data replication appliance.
 13. The appliance of claim 9,wherein the secondary data replication appliance comprises: means forreceiving the calculated difference from the primary data replicationappliance; a second buffer; means for reading the immediately previousversion of the data from the second buffer if the immediately previousversion of the data is stored in the second buffer; means for readingthe immediately previous version of the data from a storage device ifthe immediately previous version of the data is not stored in the secondbuffer; logic for recreating the current version of the data from thetransferred difference between and the immediately previous version ofthe data; and means for storing the recreated current version of thedata on the storage device.
 14. The appliance of claim 13, wherein: thecalculating logic comprises bitwise exclusive-OR logic; and therecreating logic comprises bitwise exclusive-OR logic.
 15. The applianceof claim 9, wherein the secondary data replication appliance comprises:a third interface for receiving the calculated difference from theprimary data replication appliance; a non-volatile memory; means forreading the immediately previous version of the data from a storagedevice and storing the previous version of the data in the non-volatilememory; logic for recreating the current version of the data from thetransferred difference and the immediately previous version of the data;and storing the recreated current version of the data in the storagedevice.
 16. The appliance of claim 15, wherein the secondary datareplication appliance further comprises logic for performing aread-before-write operation on the immediately previous version of thedata stored on the storage device, whereby immediately previous versionof the data is overwritten by the recreated current version of the data.17. A system for copying data from a primary storage facility to asecondary storage facility, comprising: a primary data replicationappliance, comprising: a first interface for receiving a current versionof the data from a host device, the current version of the data beingreceived substantially simultaneously in a primary storage controller; afirst buffer for temporary storage of the current version of the data;means for determining if an immediately previous version of the currentversion of the data is stored in the first buffer; logic for calculatinga difference between the current version of the data and the immediatelyprevious version of the data if the immediately previous version of thecurrent version of the data is stored in the first buffer; and a secondinterface; and a secondary data replication appliance, comprising: athird interface operatively coupled to the second interface of theprimary data replication appliance to receive the calculated difference;a second buffer; means for reading the immediately previous version ofthe data from the second buffer if the immediately previous version ofthe data is stored in the second buffer; means for reading theimmediately previous version of the data from a storage device if theimmediately previous version of the data is not stored in the secondbuffer; logic for recreating the current version of the data from thetransferred difference and the immediately previous version of the data;and means for storing the recreated current version of the data on thestorage device.
 18. The system of claim 17, the primary data replicationappliance further comprising means for compressing the difference beforetransferring the calculated difference to the secondary data replicationappliance.
 19. The system of claim 17, the primary data replicationappliance further comprising means for encrypting the difference beforetransferring the calculated difference to the secondary data replicationappliance.
 20. The system of claim 18, wherein: the calculating logiccomprises bitwise exclusive-OR logic; and the recreating logic comprisesbitwise exclusive-OR logic.
 21. The system of claim 17, wherein thesecondary data replication appliance further comprises: a non-volatilememory; means for storing the read previous version of the data in thenon-volatile memory; and logic for performing a read-before-writeoperation on the immediately previous version of the data stored on thestorage device, whereby the immediately previous version of the data isoverwritten by the recreated version of the data.
 22. A computer programproduct of a computer readable medium usable with a programmablecomputer, the computer program product having computer-readable codeembodied therein for copying updated data from a primary storagefacility to a secondary storage facility, the computer-readable codecomprising instructions for: receiving a current version of the data ina primary data replication appliance from a host substantiallysimultaneously with receipt of the current version of the data in aprimary storage controller; determining if an immediately previousversion of the data is stored in a first buffer in the primary datareplication appliance; if the immediately previous version of the datais stored in the first buffer: calculating a difference between thecurrent version of the data and the immediately previous version of thedata; and transferring the calculated difference from the primary datareplication appliance to a secondary data replication appliance; and ifthe immediately previous version of the data is not stored in the firstbuffer, transferring the current version of the data to the secondarydata replication appliance.
 23. The computer program product of claim22, further comprising instructions for: receiving the difference fromthe primary data replication appliance at the secondary data replicationappliance; determining if an immediately previous version of the data isstored in a second buffer in the secondary data replication appliance;if the immediately previous version of the data is stored in the secondbuffer, reading the immediately previous version of the data from thesecond buffer; if the immediately previous version of the data is notstored in the second buffer, reading the immediately previous version ofthe data from storage at the secondary data replication appliance;recreating the current version of the data in the secondary datareplication appliance from the immediately previous version of the dataand the received difference; and storing the current version at asecondary storage controller attached to the secondary data replicationappliance.
 24. The computer program product of claim 22, wherein theinstructions for calculating the difference between the current versionof the data and the immediately previous version of the data compriseinstructions for applying a bitwise exclusive-OR to the current versionof the data and the immediately previous version of the data.
 25. Thecomputer program product of claim 22, the further comprisinginstructions for: receiving the calculated difference from the primarydata replication appliance at the secondary data replication appliance;reading the immediately previous version of the data from storage at thesecondary data replication appliance; storing the immediately previousversion of the data in a non-volatile memory in the secondary datareplication appliance; calculating the current version of the data fromthe immediately previous version of the data and the receiveddifference; and storing the current version of the data at the secondfacility.
 26. The computer program product of claim 25, wherein theinstructions for storing the current version of the data compriseinstructions for overwriting the immediately previous version of thedata.
 27. The computer program product of claim 25, wherein: theinstructions for calculating the difference between the current versionof the data and the immediately previous version of the data compriseinstructions for applying a bitwise exclusive-OR to the current versionof the data and the immediately previous version of the data; and theinstructions for calculating the current version of the data from theimmediately previous version of the data and the received differencecomprise instructions for applying a bitwise exclusive-OR to thedifference and the immediately previous version of the data.
 28. Thecomputer program product of claim 22, further comprising instructionsfor compressing the difference before transferring the difference to thesecondary data replication appliance.
 29. The computer program productof claim 22, further comprising instructions for encrypting thedifference before transferring the difference to the secondary datareplication appliance.
 30. A method for deploying computinginfrastructure, comprising integrating computer readable code into acomputing system, wherein the code, in combination with the computingsystem, is capable of performing the following: receiving a currentversion of the data in a primary data replication appliance from a hostsubstantially simultaneously with receipt of the current version of thedata in a primary storage controller; determining if an immediatelyprevious version of the data is stored in a first buffer in the primarydata replication appliance; if the immediately previous version of thedata is stored in the first buffer; calculating a difference between thecurrent version of the data and the immediately previous version of thedata; and transferring the calculated difference from the primary datareplication appliance to a secondary data replication appliance; and ifthe immediately previous version of the data is not stored in the firstbuffer, transferring the current version of the data to the secondarydata replication appliance.
 31. The method of claim 30, furthercomprising: receiving the difference from the primary data replicationappliance at the secondary data replication appliance; determining if animmediately previous version of the data is stored in a second buffer inthe secondary data replication appliance; if the immediately previousversion of the data is stored in the second buffer, reading theimmediately previous version of the data from the second buffer; if theimmediately previous version of the data is not stored in the secondbuffer, reading the immediately previous version of the data fromstorage at the secondary data replication appliance; recreating thecurrent version of the data in the secondary data replication appliancefrom the immediately previous version of the data and the receiveddifference; and storing the current version of the data at a secondarystorage controller attached to the secondary data replication appliance.32. The method of claim 30, wherein calculating the difference betweenthe current version and the immediately previous version of the datacomprises applying a bitwise exclusive-OR to the current version of thedata and the immediately previous version of the data.
 33. The method ofclaim 30, further comprising: receiving the calculated difference fromthe primary data replication appliance at the secondary data replicationappliance; reading the immediately previous version of the data fromstorage at the secondary data replication appliance; storing theimmediately previous version of the data in a non-volatile memory in thesecondary data replication appliance; calculating the current version ofthe data from the immediately previous version of the data and thereceived difference; and storing the current version of the data at thesecond facility.
 34. The method of claim 33, wherein storing the currentversion of the data comprises overwriting the immediately previousversion of the data.
 35. The method of claim 33, wherein: calculatingthe difference between the current version of the data and theimmediately previous version of the data comprises applying a bitwiseexclusive-OR to the current version of the data and the immediatelyprevious version of the data; and calculating the current version of thedata from the immediately previous version of the data and the receiveddifference comprises applying a bitwise exclusive-OR to the differenceand the immediately previous version of the data.
 35. The method ofclaim 30, further comprising compressing the difference beforetransferring the difference to the secondary data replication appliance.36. The method of claim 30, further comprising encrypting the differencebefore transferring the difference to the secondary data replicationappliance.